{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Deterministic Policy Gradients (DDPG)\n",
    "---\n",
    "In this notebook, we train DDPG with OpenAI Gym's BipedalWalker-v2 environment.\n",
    "\n",
    "### 1. Import the Necessary Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import random\n",
    "import torch\n",
    "import numpy as np\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "from ddpg_agent import Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Instantiate the Environment and Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('BipedalWalker-v2')\n",
    "env.seed(10)\n",
    "agent = Agent(state_size=env.observation_space.shape[0], action_size=env.action_space.shape[0], random_seed=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Train the Agent with DDPG\n",
    "\n",
    "Run the code cell below to train the agent from scratch.  Alternatively, you can skip to the next code cell to load the pre-trained weights from file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 100\tAverage Score: -95.05\tScore: -96.954\n",
      "Episode 200\tAverage Score: -94.32\tScore: -97.25\n",
      "Episode 300\tAverage Score: -99.00\tScore: -52.171\n",
      "Episode 400\tAverage Score: -87.70\tScore: -107.74\n",
      "Episode 500\tAverage Score: -94.32\tScore: -108.28\n",
      "Episode 600\tAverage Score: -89.52\tScore: -45.629\n",
      "Episode 700\tAverage Score: -83.71\tScore: -99.034\n",
      "Episode 800\tAverage Score: -95.23\tScore: -108.90\n",
      "Episode 900\tAverage Score: -88.29\tScore: -99.427\n",
      "Episode 1000\tAverage Score: -86.88\tScore: -97.65\n",
      "Episode 1100\tAverage Score: -100.05\tScore: -103.10\n",
      "Episode 1200\tAverage Score: -100.92\tScore: -114.93\n",
      "Episode 1300\tAverage Score: -85.33\tScore: -37.8516\n",
      "Episode 1400\tAverage Score: -86.39\tScore: -101.15\n",
      "Episode 1500\tAverage Score: -85.79\tScore: -109.48\n",
      "Episode 1600\tAverage Score: -63.56\tScore: -32.252\n",
      "Episode 1700\tAverage Score: -56.56\tScore: -123.10\n",
      "Episode 1800\tAverage Score: -103.19\tScore: -42.020\n",
      "Episode 1900\tAverage Score: -118.08\tScore: -114.97\n",
      "Episode 2000\tAverage Score: -98.65\tScore: -134.129\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEKCAYAAADTgGjXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXeYHMXR/791e0knnXLOGaGA0iHAIAFCKCCTg8Ay0a+Bnwmv4TUgLGMyJphgDAZkDAYMCAwGZAuUQELCJJ1AOUucMtIph5Mu1u+Pmd2b3Z2ZndmdtLv1eZ57bndS1/bMdHVXVVcTM0MQBEEQkiHHbwEEQRCE9EWUiCAIgpA0okQEQRCEpBElIgiCICSNKBFBEAQhaUSJCIIgCEkjSkQQBEFIGlEigiAIQtKIEhEEQRCSJtdvAdymZcuW3LVrV7/FEARBSCsWLVq0m5lbJTou45VI165dUVpa6rcYgiAIaQURbbJynJizBEEQhKQRJSIIgiAkjSgRQRAEIWlEiQiCIAhJI0pEEARBSBpRIoIgCELSiBIRBEEQkkaUiCAIQoZQVVOHd0u3wMtlzzN+sqEgCEK28Nzc9Xj203VokBfCuQPbe1KmjEQEQRAyhN2HKwEAB45We1amKBFBEAQhadJOiRDRWCJaQ0TriWiS3/IIgiAEDe88ImmmRIgoBOB5AOMA9AVwBRH19VcqQRCEYEA+lJlWSgTAMADrmXkjM1cBmArgfJ9lEgRBCBYeRmelmxLpAGCL5vtWdVsURHQ9EZUSUWl5eblnwgmCIPgJ+TAUSTclYglmnsLMJcxc0qpVwjVVBEEQMgrxiRizDUAnzfeO6jZBEISsh3zwiqSbElkIoBcRdSOifACXA5jms0yCIAiBwkOXSHrNWGfmGiK6GcBMACEArzDzCp/FEgRBCAR++ETSSokAADN/DOBjv+UQBEEIKnvUmetekG7mLEEQBCEBz3623rOyRIkIgiBkCF76QsKIEhEEQcgQ6nzQIqJEBEEQMoDNeyrw5jebPS9XlIggCEIGMG/tLl/KFSUiCIKQAfiRfBEQJSIIgpCR3P7uYk/KESUiCIKQCcTMNPzXd95khBIlIgiCICSNKBFBEAQhaUSJCIIgCEkjSkQQBCEDkOgsQRAEIe0QJSIIgpAB+JEGHhAlIgiCIKSAKBFBEIQMwI+lcQFRIoIgCEIKiBIRBEHIAMQnIgiCICTNjgPHfCk3cEqEiJ4gotVEtJSIPiCipur2rkR0lIgWq38v+i2rIAhCUHj203W+lBs4JQJgNoD+zHwCgLUA7tbs28DMg9S/G/0RTxAEQQgTOCXCzLOYuUb9+jWAjn7KIwiCIBgTOCUSw3UAPtF870ZE3xPR50Q03OgkIrqeiEqJqLS8vNx9KQVBCDSb91RgyvwNfouRkeT6USgRzQHQVmfXZGb+SD1mMoAaAG+q+3YA6MzMe4hoKIAPiagfMx+MvQgzTwEwBQBKSkq8X7leEIRA8fO/fYPNeytwydBOaN4w329xMgpflAgzjzLbT0TXAPgpgLOYmdVzKgFUqp8XEdEGAL0BlLorrSAI6c7hSsVCrjYngoMEzpxFRGMB3AngPGau0GxvRUQh9XN3AL0AbPRHSueZ9P5SnP/cF36LIQiCYIvAKREAzwEoBjA7JpR3BIClRLQYwHsAbmTmvV4L9+LnG9B10nQcqayJ27e/ogr7jlRFvu89UoWy3Ud0r/P1xj1YtGlf5PvUhVuwZOsB5wUWBJWa2jo8M2ctDh2r9lsUzwmPQKYu3IKa2jpsKD+MrpOm48sNu32WLP3xxZxlBjP3NNj+PoD3PRYnjje+2gQA2FdRhYYFuVi29QDaNS1Ey0YFGPTAbABA2aPjAQDDH/sMR6pqI9+1XD7l66hjBcFtpi/bgWfmrMOew1V48IL+fouTkPJDlThcWYNuLRtGbT9WXYuVOw5iSOdmtq/5xMw1eGLmmsj3j5ftwE96tExZ1mwmiCORwPLFut3Ytv9o1LZzn/sC4/60QPf4I1W1XoglCJaorKkDAByt9v65rKmtw7Ql2235JE56ZA7O/OO8uO33frQCF/3lS2zeUxF/kmH5+uVWyDuaMqJEbPDzv32ju738UKXHkgiCcxw6Vo2hD87GOX9aEGWOdZKX5m/ErW9/j38v3WH5nDoDfbNsm2L2PWjRLFdXxzikY34GgH99ty1yTJ1RgYIpokQEIcspLduHPUeqsHLHQTz7WXTqjKqaOhzTGbls2nMEW/ZaHwls3acc64Q/pk4dzeRYzDj4wueJ54d0/+3HuPjFL1OSK1sRJSIIWUKlqgxKy6LjUY5U1ffSG+SFovad/sRc9LlnRty1Tn9iHoY/Ptdy2W9/uwUAkB9KvckJW8RyLF5qzqqdpvs3lB8GAHy/eX8qYvnK1G83+1a2KBFByHDKD1Vi5oofcc9HKwAAZTG+hJCmRx+rRBJlhg03wGHmrt6Fd0u3GB6fn5t6k2N3JHLomL4pK8xZT36eskx+M+lfy3wrW5SIIASAX/x9IbpOmu7Kta965Vvc8MYiw/2kaYyJgNo6xh8+WWXJXHXWk5/jR42iufbvC3Hne0ujjlmoGfmU7bZuAjOiXoko3w9UVGPTHiWUfn9FFVbtOBhlgsvNsb7QxhMzV6Omti5p2e75cLlr9zGoiBIRhADw6epdrl178x79uUphQjlaJULYvLcCL32+Edf+faGl65+rM0l29+H6YJMX5tX7JJ6esxZdJ02P2maXsDkrrPzG/mk+Tn9iHgDgkhe/wrg/LUCfe2bg42WKE3/1j4csX/v5uRtSmq/1xtebkj43XRElIghZTmxHPRyGu37X4ahtT8xcja6Tpsf1tPWiE8c+Mz/yuUYn6umxGaujRihhnp69Fsu36TfiS7bsxxfrdseZs7QmN63Mn63ehTkrzf0hetgYuBiSTelVRIkIQpaTY6HVPP2JeXh+rvnoQTuHavfh+lDh2jp989ClL34Vt+1Pn67DT/+sn/7n/Of/i5//7ZtI6G+s1P+M8cW8t2gr/ud1+6n1nIj0PVadvEks3RAlImQEB49V25p8JtSTE+MT0WtDNyfwjyzatBenPvqZ7j6jiX5hdh08FuVXSYSRY/2OGF9MstQ6oEX8mNDpF6JEhIzgguf+ixFPWA85Fepxwnyzdudhw31mjfLslTsx7JFPcfIfPrVcVr1PBK5MEEzFsR7GCUWULogSEQLJgYpq3DdtBSprrPXoNhokusxWqtVkixVV5uGtsRAITpvz9XwiYaYt2R75bNWPEB6JEAF/mbc+NeF0MJPXKnVZ5BMJXAJGQQCAP3yyClMXbsEJHZvgoiGyQrJd3lu0Fc/MWYejVbVRIbx6aNs7i1MvbGHWKw8lUV64gWYG/jhrbbJiGVJj4MOxQ/g3b9lbgaZFeSguzEv5mkFFRiJCIAlPEHt8xpoER2YW1Q6YUoD62el6KUsSY78XbdbxNuvZV2v2We28h0/5bvM+0+OuH9HddL+RGa86gQ/HCmElMvzxubjg+f+mfL0gI0okAGRTOKBVwo7JHw9ad7hmAv9Zuj3xQTYwerKOVdfi5QUbUVvHUce4MBAx9VtM1yRktPoWhN+XSpMIqD5ti/F/o3uby2VQYHKKNxrt6GtDeWabWkWJCIHkqCZFd1VNHa5+5VvD+QOZhFnD6CRPz1mLh6avwrQl26K2EwH7K4K9aFW4fa5OYHYijUrs2KyB5etv3596x6U2izqGokQCQBY9b7ocPFaNnTEjDq1jctWOg/h8bTlunfo9jlXXmjrb031U51Xjc/CoYi6sqKqNq7NLdOZveIFdx3qiCCituap1cYFlOYzmtdghdvR12CAVfSYgSkTwnVFPfo6THkkc4rmx/Aj63DMDx/0uPqtsmHSPrHQqZDWRM93IeESuGLSscTBBosRYEs0/0dbBwxcOiNvftUWRrfLs8M9FW6Nm9r/1zaaMXa9ElEgAyMxHyzq7HFzUK93j852Sv3RTvdPZjjpJNjrrtx+knkX2HIMVQo1IFEWlHYkUF8YHor57wym65zkxGJwyf2PU90c+Xo3n5jofjhwEAqdEiOg+ItpGRIvVv3M0++4movVEtIaIxvgpp+AvT85agxveKMXMFT9GbU+2EV60aW8gTA5OzFEAgH8vseagn/zBcuwOyMqc2iCKvZoVFntP/kTX2Z2orhKNxlo3LjTct2zrAdz2zmJHOyUzlv+Y+KA0JKjzRJ5m5j9qNxBRXwCXA+gHoD2AOUTUm5nTPr9Autvx/eDPnym9upkrduJ344+PbE/Gp3CgohoXv/AVzjiuFf5+7TDHZEwGpyepWbncywt+cLRMJxjy4OzI56raOt11TWpthOImNu/VU1lTF8lM3LN1I9x0Zk/L52YjgRuJmHA+gKnMXMnMPwBYD8DfN14IBA9NXxX5HLtqnxXCjvoV2w86JpMey7YeiApp1SPVaSIHKqqx48DR6I3+uTlcRW9OzXs36puoYpn56xGG+w4crY9Oe2Jmds1TSoagKpGbiWgpEb1CRM3UbR0AaNN0blW3xUFE1xNRKRGVlpeXuy1rysg4xDmueVVZA+OGN0rx7kLjFfbC7DtSFZm0pk1pfuhYNW59+3u8u3ALFqwrxwFN2OvXG/fEZYwFgB92H8HSrfuxcvvBuGgzQFl346a3vgOgKK6nZq/FSY/Mwbw19WuJpDoS+cmjn+KUP+gnQrTCp6vcW9ckFfRG689+Fu1jMJtcGKtHj2tbbHhsKjPW3ZjxH3R8MWcR0RwAbXV2TQbwAoAHobStDwJ4EsB1dq7PzFMATAGAkpKSrGqjfzxwDA3yQsjPzQGDUZSfi8qaWuw+XIW2jQujFiCKZc/hSjQsyEVhzBKpAHCksgavf7UJN4zobpo6fH9FFbbtP4oteyswum9bPDd3PS4c3AGdmiuRMCMen4u9R6rw1d0j0aggF/dOWxF1/lvfbMZvP1iGgR2bJFkDCjNX7MTMFTtx2YmdAChpxCcO6xz5HmawxmwCKJPfxp/QDgPumwUgOrdT2aPjUVNbh8unfA1AyRr7l4lDcM6Adli6dT/Oey56ZvL6h8dh2/6jGP30fFTW1DdMQx6cjf4dmmD+WqWD87cv6s1JRjb4rfsqUJSfi+YN83X3V9bUIj+UgyNV1qy72jZZO93wq417LJ0fRM4b2N5yrjUzEkV9mZEfyom619mAL0qEmUdZOY6I/grgP+rXbQC0LUBHdVva45QZ/B9fb8LvPlwete39//cTXPzClwCUCVcf3XQqXvtqE64Y1gltigvx5jebMKhTM5TtOYJb3v4eQ7s0w3M/G4yPFm9HHTOOa1OMrzbsQWVNHd74ehMem7EaXVsUYdTxbfC7n/YFoISldv/tx3HyfHTTqXhq9losLNuLFg3z8eHi+gb5jzPXoE+7xnj9q/qV4Jg5EuVjtTG0ypIt+7Fky/44JRLLTW99h/EnjDfc/3iMeeOl+Rsxrn/bOAUCKAsvdWpeFNeo7D1SFVEgQHSjVWXQAJ322Fzk5+Zg7UPj4vbtO1KFwQ/Oxl1j+xjKne5ofRpG74vZKCDRCGFgxyaRFQ1TCW7o0qLINKNxJhI4xzoRtWPmsOH4QgDhVnEagLeI6CkojvVeAL51Wx5mxvNz1+PiocFPAhirQABEFAgAbN13FEMfmgMAePbTdWjZqCBqGVMAWLRpn6455Cc9WkQ+l+2pwMtf/ID3v9uKd244BYeO6c9w3nNEufaCdbvj9n21cQ9e0ygQILoXbmddbC+JXY1vyZb9ERNaLH9d8ANGHd/G1vUPGtQlYKxgdh5STGcffq/fpzqkM/9iU5qtveJ28EleqN6yv2pH8r6xpg30R4qZTOCUCIDHiWgQFHNWGYAbAICZVxDRuwBWAqgBcJMXkVlrdh7CH2etxdw17vlW2CevSKwCMUOvJ7evohqvfVmm20glQq+3pu0A2lkX2yumLdmu28B8vtb42Zizyt7yrAePWks58vKCjRjTr23ETAjYe460Ziu3gwO98hMoEyX1C9NOovz41uFx+5+eMAgvL9iI177aFMhnL8gEzrHOzFcy8wBmPoGZz9OMSsDMDzNzD2Y+jpk/8UKesKnhqMPmlWzAbuPklzK1yq1vf+/Ksqfa323FkvLR4m14aPoqXPm3bwCYzzIPep1aRWvOOmCgaK0qq77tG8dt69S8CPee2y8p2bRkSn3bIXBKJGiEG8IcF2sqU6eJ2FYiGVoPTrJ+1yH879TFAOLzMSVbf+lQ7amas6womGyMrHICUSIJMFrPWVAwWpNbsI6d9vHA0XjToTyaCkT+10U2doSC6BMJFPVLcUY/ndn4sBhh1EuUKrJGKvX05Kw12KiuV1EhJldDRM+6hyiRBIRt1PIQ2seuCcIpxZxNaWT+rJlwt23/0bj9VqoiHerLStgspZiD2E5qFCOCX5POI+ashITNWTp7HHr50uAdNoQ5O18cR5EKTMiN/1iU8Bi/TVlAeihkpxElkoDwSCTWJ5KFz4oxBnVht4qcimxJt3uj/d1pJnr64KGCObFrs8QHZRCiRBIQXkhGz7HumPklDZoOo99qJrlf0VnBr81gkSn1RXDGJJUKDOimDcpkRIkkIBK3H/NspkPD7xVBr4t0MjEEwCKTkfi5YmOmI0okAcwmPhHHynDoQi5i1MFjNpPfpmPd1tEm10mHCtUQnQzRDwH8KNR5guET8VsC7xElkgAvfCLp/twZmrpsm7Mc8omkKIfXaMX795LtGPvMfFeunfkYjze8VDB+m9S8RkJ8ExA21eg9F+nW43UH5+rA6Eq2HfQxJ6TbXZLcTc5x68ieGNCxqWflmT1rqaxTEmREiSSAjUYijpaRbs1cNEY+kaD8qqDXr9/yBbt2rKPX0bt99HEAlLVyPJPDYHumpogXc1YCtu5TJnDpveeZ8vKlgplPxKnoLNuTFtPszqSXtMHGd0tSwDssbiBKJAHPfroOALD6x+gU4E72HtPhsXNKUZgX4tBl0tycJSSHmf7w0k/huyLzGFEiCbisRFmM6oJB8cu5Z2GnIw6zBIyOTR5M9fyA3ye/5fPbnJZJZGNNihJJQLgH07Ag2n3krE/EwYt5jKPmLJdewaCbt/yWzu/ynYLIOHuWl4ODLBuIiBJJRFWtElGRnxtfVUFvnJzEeMY6w6lmyG3fSmBxUeC0q4sUCELjnU31HUaUSAKq1XWtXc2dlaEPnl9eo1jlHvQX28pqhm4S9PpxgkyeJzJrxY+elheLKJEE1KhvuN6oIxtevjBmoy7noqqcId3ui98jWr/Ld4pALErlQ12+t2ir52VqsaxEiOg0IrpW/dyKiLq5IRARvUNEi9W/MiJarG7vSkRHNftedKP8WGrDSiTu2XCynx38l9jM1OR2upJUlULQlYq78iW+uNv1E4QZ3F7mzvL/13qLpcmGRHQvgBIAxwF4FUAegH8AONVpgZh5gqbcJwEc0OzewMyDnC5TSIxTs8m9IIgymeFmI25tUSr3yveSICRZzJS6tIPVGesXAhgM4DsAYObtRFTsmlQASOm+XAZgpJvlJIujubPS/MFzK+dVou2G14mRJ+gjvWBLlz6YDng89Yl4V1YQsGrOqmLlzWQAIKKG7okUYTiAncy8TrOtGxF9T0SfE9FwoxOJ6HoiKiWi0vLyckeE0Wso073xt4XJbzVs/H2Kqkq7BIw+R2fJPBHnyMaqtDoSeZeIXgLQlIh+CeA6AH9NtlAimgOgrc6uycz8kfr5CgBva/btANCZmfcQ0VAAHxJRP2Y+GHsRZp4CYAoAlJSUpHRbzfJCBWUynRcY1oPZPJEUo6q0ZaRC0OvXVXOWFZ+Ie8UHBm9HB94ORfy+f5aUCDP/kYjOBnAQil/k98w8O9lCmXmU2X4iygVwEYChmnMqAVSqnxcR0QYAvQGUJiuHHbKxh2GFZKK2bJdhN8orze6Vm+a2bPKJAP6bkjKoKi2TUIkQUQjAHGY+E0DSisMmowCsZuZI7BoRtQKwl5lriag7gF4ANrotSNhZp2ci8auR9AMzEZ0yZzkX5hUrR7Dr192RiJVjgl0/VjFTIJ7OWPdYkfntgkmoRNRGu46ImjDzgUTHO8TliDZlAcAIAA8QUTWAOgA3MvNej+TJqN5aMpj9fKNGus6neSKx5Qb91tmtJztk20jEb/zosPh9+6z6RA4DWEZEswEcCW9k5lvdEIqZr9HZ9j6A990oz1QWQ5+Ic/03vx8CKxi+HEmMUGyXneLxQW8k3RTPSqMW8OqxhWHuLC+z+HpWUjCwqkT+pf5lLZky5Hca01pxKDrLr2V2PcNnc5bbpN39SDP8VlpWHeuvEVE+FEc2AKxh5mr3xAoecWtUsIPzI9LgHTM3Z+lvt2/OcqYi4nJRBbx+3TVnWZmxHvAKcoBM9on4ffeszlg/A8BrAMqg3I9ORHQ1M893T7Rg4feN8ptk2hm/6izdJhu6iZXkjlmgQzwjG+vSqjnrSQCjmXkNABBRbyiO76GmZ2UAnuSMSuNGjtnYO+TY8ri255ukJofXuOoT8bn8oOBpFl/fDUzeYnXGel5YgQAAM6+Fkj8r44m8YEFviVzGLCWJc+YsuzsMDk+zW5Xp5qx0ux+pkM4dwmSxOhIpJaKXoSRdBICJ8GiSX1CIN7Ozcy9HOjx3Jj82icAtW9hdbyPdQnz9nieSDXiaxTe7BiKWlcj/A3ATgHBI7wIAf3FFooARfsFTedGZORDpsN3CsPdle6a5cTi1PXmsXTcouCqeFZ+Ii8Ur1w92/TtJwB81V7CqRHIB/ImZnwIis9gLXJMqUJjY+52ase7MZVwlmVnpdkcQToX41tWl20jERXNWANYT8Qqzjpq3Kxt6V1YQsOoT+RRAA833BgDmOC9O8IiMRFJoihK9pOnwEpuZrJzyiRiW7chVgoubv6+uzsIx4hNxjCz6qRGsKpFCZj4c/qJ+LnJHpGBhZs7KqmG6WaJFo+1OjdRSTMAY9EbM7yy+WfQYe4JEZ+lzhIiGhL8QUQmAo+6IlHkkekczVRnZ/VVOOejjHevBrl83RwJ2TYpuEAARPCPo/jc3sOoT+TWAfxLRdvV7OwATTI7PGMINkN7cg2x6Xoz9FcYTZmyPIAwvZOsy8YcH/D45FAitf7QMRLzH6xnrPt9A05EIEZ1IRG2ZeSGAPgDeAVANYAaAHzyQz3ecis6yUkaQMU8F78xkQ7vXNzw+HSpUg7vi+j9PJAh45ezO/JqMJ5E56yUAVernUwD8FsDzAPZBXTkw0+HI/3gTSTY+MLGY1YH9xt/ediNiTTjBv0/WJfRryWHBOl57RPyOBktkzgpp1uyYAGBKOCU7ES12V7Rg4MRLmNgnkjplu48kPigFzGaTG0dnOVOG/VDh6BOmfrvF3gU8xs4zZtvP5MI17RKEkY5nzm4ffqrf1ZtoJBJSl6oFgLMAfKbZZ9WfkhnoRPwE4eUIc8Yf57l6fbPfGrjoLM3nI5U1eHrOWmcEcQlX5xpaqDu3Q3yzjUyeWKxHIkXwNoDPiWg3lGisBQBARD0BeLXKoa8YOdaNtuleI+E8kcx8iZ3yZaQS5VWbBnVr5/7bNme5cE27BOEOiE/EPUyVCDM/TESfQonGmsX1T3sOgFvcFs5PqmrqcMHz/8WeI5UA4l90U18AM2o0Npjq2jrkhShjeyiGjb9DDZ59n0j9CbGz14OIHQntdjgspYK3dcUkCP4tcJTMfMuNsbLG+tc624JtH3CA7fuPYuWOg4b7L3j+v1Hfu06aHvnc7e6Po/b1u3em4XW052m//+GiAbhiWGcAQEVVDaprGE2K/EucbJam3bjxd6r1SN5BX10b/BbMjqKzPyrz//d3bdkQG1322SXCq4Y9CPXtNVYnGzoOEV1KRCuIqE6dvKjddzcRrSeiNUQ0RrN9rLptPRFNclO+nJhRg9fPxrul9c7gM/84DwMfmOWtADGYhfE6t6yt/vZUsvjWxpw8f205Kmtq0XXS9DgF7hcHj9Ukdd7uw1WJD7KCy892y0b57hYQMDLU4GCIn87x5QAughJGHIGI+gK4HEA/AO0BzFEXwQKU8OKzAWwFsJCIpjHzSjeEy4lRry9/ET8t5oNf/QQndGyKI1U1aFyYB2bGd5v3YWiX5gAQlb33WHUtCnJzQEQ4cLQaRfkhVFTVgplRUVWLVsUFyM0h1NQxek3+BN9v3o/HZqzGnsOV2HlQMamNf3YBdh+uRJMGeSjbXYGf9GyBeWvK3fj5jmDfkuSMWWzxlv2RzxvLD0ftu+qVb3H+oPZ2BQsMqZj2DK/pkhZZv+sQerYu9j16CPDO2R2An+o5vikRZl4F6N7c8wFMZeZKAD8Q0XoAw9R965l5o3reVPVYd5SIhYducOdmAIDGhYqZiYgiCiT8PUxhXijyuUmDPPW/oqmaarKQ5YXqz3lh3oao8lZsV8xrYaUSBAVitsKjY/NEbF7ndx8uj3z+2cvfxO3/aPH2uG3pwm/+ucTW8ZZmrLvU8o16aj7KHh2Pfy7a6k4BASXLBiL+mbNM6ABAG9i/Vd1mtD0OIrqeiEqJqLS8PLmG1ooS8Yp7z+3rtwjmowqHHOs2L5+VbNtvL2WdlbqrcTH44IopcS5V1zB7Y73ziXhUUHSpfhQawdWRCBHNAdBWZ9dkZv7IrXKZeQrUGfUlJSVJ1bCZDmnZKN85e7QFrj21G649tVvEhv/4xSfgshM7AVDMZH3umeG6DGYLRh2r1s83nsr8Di23vP29retkGjW1FvK5G/DVxj0OSpJ+5fuB11GYfneyXFUizDwqidO2Aeik+d5R3QaT7Y5jdmM+v+NM1PgY9XNpScfIZ62ZzE02lOtH18xcsdPwnGc/W2+rDDdehsK8HEMl5xfHt2uM2ro6rN15WHd/z9aNor6PeupzL8TynVTvVYO8EPajWnefd/NEsm/YHERz1jQAlxNRARF1A9ALwLcAFgLoRUTdiCgfivN9mltCmD0MDQtyfQm3XfPQWKx7eFxcT2fO7ad7LosbjHlmvuPXnPebMzG4c1Nb53S/O7mordevG5b4IAC/Gd0bT102yHB/eAT3bukWdJ00HWV7KpKSJ52Y+5szIiHtZtxnYtpt1jBxFNipPVvYkisZgmMI9wY/Q3wvJKKtUBLjkvnKAAAgAElEQVQ7TieimQDAzCsAvAvFYT4DwE3MXMvMNQBuBjATwCoA76rHuoJRr/iusX3cKjIhBbkh5IXib1lszzUbmX7rabrb83Nz4sJ8E+GUi+COMcfhsYsHRG27fkR3nHV8G/Rr39jQ1xUu/jmbI7l0x8p9Oq1XK7x34ykozLPXdBER5tx+Ov56VUnig1PAbdPS4zNWo7Rsb9Q2v8c+vikRZv6AmTsycwEzt2HmMZp9DzNzD2Y+jpk/0Wz/mJl7q/sedlO+vUe883k4ybBuzfHMBONebjrSoWmDuG23n90b153aDQBw7ald0a99E5Q9Oh5lj45Hn7bFkePyQoSqGnMTyei+bXS3X31KF8NzxvVvi9tG9Y7b3ruNotDHD2iHn5/cBRNO7IwbT++BUE50/5SIcK0qPwAsuXc0np4wEK2KCyKtgjZSL9MhWFMiOQSUdG2Oj28dDgDo1rIhFtx5JhbceWbCc3u2boSifA8CUh26bat2HMSb32yK2vaXeRtwyYtfRb4fPFaNz1bvcqbAJMmuJIo2uOO9pZHPxYW5OKROCAu6zfOFiUNSjrbp3LwIm/emZkKZddsIjH7aGfNUbERS2aPjI5/vPqcPcmMa6PxcpW/UuXkRGubn4pKhHfHQ9FWG188hwr3n9sX9/46PFv928ln4n9dKsXRrfKq43FCsYgCm3XwaKmvqImHcADBpXB80K8rDHz5ZbShDo4JcXDi4Iz5bXY4V25Sy9EadYdo3KcT2A8cM96cjvx7VG29+s9n0mLAyDpt065jRqXlwVup2ciQy7k8LAACfLPsRr183DDma57ymtg65oRzcNtX/ZOpB9IkEggMV9SORto0LccPp3T2X4Vdn9MCs20bYOoeIUJCb2m195RrjIX/fdo0tXaOXgya2hvn1wQP9O0SXnxfKifMRndG7FQDgH784CTk5hF+c1g0bHjkHax8ahw2PnBN3/VAOYVCneL8JEaF1cSFOV6+nJTeUE6e8mJVAB60CCRO+/kndmsftA6I7rxt3H8Hna8ux+sdDuseGZcskiKCMwhIQDr0P/3q/I5P0cDrt/Bfrd+NYTW1Uepxwp2jtLuNnxCtEiRig7cz79b7eObYPercpTnygBgJQXJiH49s1xsMX9k+q3J6t9ctc+cAY9GpjTTkQEb6dfJbpMcUF1gbC953XL/L5P7cMT3j8r0f1xud3nIHOLYoisoRyCPm5OXFmJWV//cRRPfTybxXlhZAbM1JoVmTs2D2pewssuXc0zjpe33QWfsbC0l39yreG1/KD03q29FsEAJp6Uv8HyTJwpLIG2/YfxbGa2pRSvUyZvwFz10SbqAgUZWEI7z9a5X/koSgRA2LTRRTmKr3hglxvQmqThUjpWX/yv8Mx8aQuWP3gWHx006mWz3/zf04y3FeUn2srtLl1caHu9pF9WuOB8/shFGMOWvz7syOftVE0ej17M3JyCF1aNLR+vNoixY4swujN0ygqCMX5LAZ0bGJajtnvCI8srHZYvO7YuB1QYrX3Hu4EhO9ZkEYiJ9yv5LebvnQHZt2mHzH5z9IteGq2cf7ajeWH8cjHq3HtqwujttcxR7VJ4Wf1WHVtqmKnjCgRA6JGIiDceHoP/OqMHph4UuIwRD+JfRkL80IY2Kkpyh4dj2aasOT7zu0bZ1q59tSuODVBj3N0P/2edCL+MnFI5PMr15yIq07pipd+PhSjjm8NALhiWGc0LcrH9/ecjYsGd8CLPx8aOb69jmPdSS4YrOTSim2Yw7b2ozov6ui+bZEbm2DNAb5Yt9vScUHKqGCVCzQ5y0b3bYMf/hBvWkyE34lRzQgHBgzu3BTNDcKN73hvKZ79dJ3hNUY+qT8naOu+o1GBB7k5OdhfUYXDlckl73QScawbUn/DiIAG+SHc6WN4r2Usti3nDmyPT5b/GPneoWkD3HtuvdmISP8FPX9QB6zcfhAvzd9oS6yRfRRloe3tn9S9BU7qHh2336xhPp6aMCgyV+L4do3Rv4N5Dz8VtE762HiEFmpDMKJ3qyiHb4uG+TilRwt0b9UQRdNDePWaE9HDIR9QtcXZ6QaDpkDz5GWDMKJ3K9z+rpL/S+vXsTsCCzuZg5h6fZSByVLL0apaNFB9fXsOVyKUQ2hqYg4d88x8TLu53qKwZuch3P2vZakL6wAyEjEggM+mJay+jC0aFZj+xrvHGSvM/CQc9+Gy9HwSehARyh4dj0/+N7EPxC4vXamMcmJ/x22jekV9D2f7HdOvbVQI6ctXK4EHbRoXYuUDY3FS9xZo2SixU9gKFw3pmPggpKdjPZRDkRDbRK+XUWBGKMax7vWaY8zsiOI6/vczcLRKGeEOfWgOBj0wO+E55z0XvYaRthPoJzISMWCPZp5IOr2wpknoYk0BJq/yL4d3x8g+rTHqKWfCdMNO7d8HIJlkWCk0jHHs3zyyF07v3RordxzAyd1bRNVXp+ZFUaMWt7jnp33x9y/LEh6XPk9kNFZfpff+308w8P74NXRyYnxHXjvW7//3Siws24v/3HJa1PORjFnp9a/KcMPpPRyUzh9kJJJh2FF4Zr04IuPlfLUdsQYWc3eFcggbHjkHE08ynsDnJZ2aF+narQd0bIIJJ3a25Zh3klAORSZRmqG9NXbTupjx2f95k0InUWe+SYM8NNVJLRQ+Lez789pi8Pcvy7Bi+8E4P1ky2QV2HaqM+h6URdLsIkrEAunU67OTDjvRsNzoWkEKq8xEJpmYErtowpbDlHQxDk+2S/dWjfDczwabHmO146CH0TOl11/RWzY4/MzmREYi3sDMmKcJu43NaLF1n/3JualkZw4SokQskEbWLFuyJrInWxnVpFPdpAtmPqd8dW6Km471n55gvvKjVb+WOfab/8cvOQEtwr6nsBLxaChy/79X4hpN2O1pj83FoAdm4a73lmLHgaNRUWNF+daUbDqZyc0QJWKBIwEIo7OKndmyiV4/o7ZC+9569Rq8ck0J3rn+ZI9KCz7a++x1Y2Rlyd3/OS3aJPf2L5V7V6+A4nOJxRJbymUl9StBeG3OWrRpX9y2/RXVeKd0C075w2dRnanLT7Q2DWDPkSrMCIhzPBXEsW6BPWmUjNFWe5LQnGVlJOJNAzayT3LzUzIVrzuxWhOmFSUSK98pPZRQ7hG9W+HKk7vg5pE9dc+LWvDN1GeX8BBHWbYtPneaFu1IpIHFkci/l2zHv5ek71LNYUSJWODv11pbJyLdSPQCWmmoMmNAHlw6NmuArfuUBJTFhfWvq5+mkFR6/3mhHDx4QXw6nvCv+eKukZFJdWbF1OfOCoZ/LpvfAzFnJaBby4YY6qDj0m3M2pbYfYl6lEbXijorm98eD4hKN6+peK2p0YtboB2V2m23f3WG9TDWwrxQJPTaTEFE0p7YE8U1MsW/kQyiRBJgtNZEULHlE3HgDczeV8dd7hhzHIDodPPa2+Vnm2XFnKU9xEqmB7Pfc9OZPXDp0OhJmOF6adfEnZQ4l59Y73/5fnO8PySW3YcrEx7jB16M1ESJZBhmL2Ps8+RMdJaoETe46cyeKHt0fFT9ahsEP3NnudEs6XV+wuX86oyeeOLSgVH7igvz8JeJQ/DadSe6IA3wyIX1K1JO/mB55PPr1w3DrNtGoHFhtCfALMT3P7echnMGtHVeSCAqx5xfiBJJQFCGy1Yxa1pif0uyvZSo6CzRIa4SIoORCIy+uI+lkYiD5Rg9Y+cMaGeYKdpJVu44GPk8oncr9G5TjKX3jYk6xmwBsf4dmmB0X3eUyNj+7lzXDqJEMgyzkUGs0hBzVvAxCrMe3c+/xsMNC4mpL8+Hp8xu58iZuTP6xC7Elgit7F7EHfiiRIjoUiJaQUR1RFSi2X42ES0iomXq/5GaffOIaA0RLVb/Wrsp4y+HK3HuQYn+sIqdkYiVHqX+dbQZjkWNuIl2SVTt7UqUst8Nvp18Fub95gzPykunV89oLRonuGVkr8QHaTi+rT2lkyp+jUSWA7gIQGx2v90AzmXmAQCuBvBGzP6JzDxI/XN1dXorS3UGETs+ESfeUVEh7hJtztL6ROqP8aqn3rq4EF1bWsspZlcB6P2CSJ4sHx4yu52j2FUunUSroGZbWC47aiTihkAx+DJPhJlXATpZZZm/13xdAaABERUws+ehD40LleRvdlfVc4KfndQZc1buTOpcs4c/duThxChLBiKpMe3mU7Fu52HD/UbrXgV9Uaq+7R3oDauPZ1B/alF+CBVqOne9Ge1aUsk3pzWV9bK5XLYXBNkncjGA72IUyKuqKeseMmktieh6IiolotLy8vKkCr+0pBMevKA/rh/hfarmRy4cgG8nj3K9nISTDS2dGNA3PE04oWNTXDzUeA2RHNI3ZwWNzuoqkIDSW754SAd7F9B5jIKe6POmM/Vn3TuN3RU0ve5guKZEiGgOES3X+Tvfwrn9ADwG4AbN5omqmWu4+nel0fnMPIWZS5i5pFWrVknJH8ohXHlyl6QWYAosseasJN/RoMxXyAaMGgTt5iDcg19rFvTq1aYYRIRnrxhsyfxiRPj59MOxboVfndEDf77CPONxspx5XH27lYrTPq3niTDzKGbur/P3kdl5RNQRwAcArmLmDZrrbVP/HwLwFoDMzEXiIq6Ys1K+gmCGtgGJDvENfs2fN7C9ZfOL3u958IL+KC7MddVpnQpEhLyQNdnsvmqn9apXIrkGZRhl0vC6UxGobjYRNQUwHcAkZv6vZnsuEbVUP+cB+CkU57xgg9jnuGeCdcGbGaz5PEEzm/fk7i3w+nWKPr/mJ13jjm3b2P04/kzGqEEIwugDAIrVFCVuyHPFsM5Ydt+YqAi1oFFZ486aIGTwWYuRv1Z7fMY61onoQgB/BtAKwHQiWszMYwDcDKAngN8T0e/Vw0cDOAJgpqpAQgDmAPir95KnNz8/uQumzN+Ilo0U5fD0hEH4eNkO3PX+Mt3jG+SHUPboeEz9djPaNqlXBj1aNULZo+OxftchdGxWhMI85bi6OsYlQzuif4cmkWNn3T4Ch4+lTyr9oBEyCLWJMme5UO5jFw8wfC60nNitOT5bnXqgZFCUol1cUyJR5kqbleNxZfoVnfUBFJNV7PaHADxkcJr/8/vTnIuHdMSU+RsjI4ziwjz89IT2uOv9ZVG5gmK5fJj++gg9W0ebKnJyKEqBAEqUWzjSTbBP1DwR7fwcl81ZnZv7szxwulFlUYmEV6S0StRIxGg0auuK7iGp4DOEGb8ejq837LF0rPahbFiQi/UPj3N1xq2QPEb3xeueu3aNjN5tGmFtTFhyqkotXZ8+q0pkaJfmmHXbCDTIC2H443MTHq8dfTRVzVYXxUS8PXRhfEp9AOjSvAhLtuwH4E1EnyiRDKFP28bok+RMVTcnSgmpUZhb33gbNQhuKpQGeSE8PWFglP/s3RtOwaAHZrtXaBpxUvfmlo/t3aYYuw4es3QsEbDmobFYsf0gurdqhA2PnBOXAscog/EjFw3Apr0VEUXiNtJ6ZBFBj7sX4rG6Sp7ThBVTw4JcjO3fLmpfU52Ai1QVWbqmz+nXvkkkuABA1Gc9rHbYCEBBbghDOisRWKEcslxHRXmhyBIWXrzzokSykHQIDxUUijRKxI8uQJq27Y7y0pXW3LGPXTwAMxLMizEK140jhYrP6hBfQRCiadIgD82KFJu43nyJ4b1aZkSnIMi/IDZYJA5V+NN7t0aHpuaLZOVZnH0e5PqIRXwiWUTXFg3Rvkkhfjv+eL9FESxSmBfCd/ecjec+W49Rfdvg1rfr08t9dfdINCvKx/Nz1ztebrKNWKJGNB2xWhdW9IPVkUgqowkiwjkD2qF3m2LbKVOSQZRIFlGYF8KXd5/ltxiCTYgIt5wVnw7craVho8pOsF+b9eC/k0aiuDC5JiXIZjOruahCFo6zOvv+h/Ijlo4zolvLhuhmMeNyqog5SxCEOOw6uokIHZo2yMg5QVarwkqYvNV6/WyNqytdOIooEUFIcwLcibdMkP06Cdv9SMp6535DbV36RFKKEhEEIWWCqwJSx6qCc9IkV1MrSkQQhDQm7OvYdch8PbhfjuiOovwQTunRIrUCA6yFvPDXfB3jq7QzEnnwgv740+WDnBbJMqJEBCHdcaGVW19uvNqiloEdm2LlA2PRslF6LidtRtjFkcixPuWqEpzdtw0a5ScXVPDt5LOiEpwCQE2d9cSOV57cBecP6oBBnZomVX6qSHSWIAhxHFWXfTUih4A6dk5/OXWdbi0bokXDfNw1ro8zF0TiQdIpPVqkNBJrXRy/XEJNEj6RN34xDNv2H01ajmQRJSIIQhzHqs2VSG5ODqpqU0+DTuRsksAG+SEsuudsR64VFosIePWaEz1d5fT+8/rZPqe4MA992nofHSdKRBDSHDdM9id0NDeN5OQAMNcztgiiS4Q1UVdn9mlt+/z/ThqJo1XJraWTaMG4ICFKRBCEOEq66i+9GibsJ0ijSNSkSdbUlsrs/SCHPMcijnVBEOJI5EwOz85OdT4DuzDHwmmclkw7a31sv7aRzwM0OboCXB1xiBIRhDTHjQYn0TW7tWroaNlBbjOtpj2xita3cmlJx8jn8LLVbpTpJmLOEgQhjkSN2KvXnIhFm/ZlZJqTWJxuz9s2LsTG3UpuLKN6TiMd4s9IhIguJaIVRFRHRCWa7V2J6CgRLVb/XtTsG0pEy4hoPRE9S0Ee/wqCh7hhP0+kRFo0KsBojSkmVYL8Njs9Knj12hPrvySxfvrnd5yB+Xec6ahMqeDXSGQ5gIsAvKSzbwMz602/fAHALwF8A+BjAGMBfOKahIKQxVhMNpsyTof4pgNtGtfPC0lGQXVp4U12Xqv4MhJh5lXMvMbq8UTUDkBjZv6alXwMrwO4wDUBBSELmPebM6KcuVq8HugHKRopNqrKzaowunQ6GVqC6BPpRkTfAzgI4HfMvABABwBbNcdsVbfpQkTXA7geADp37uyiqILgP3bam5aN8rH7cBUAoKtH602kE4t+NwqFedHr2jttztJeTnwiJhDRHCJarvN3vslpOwB0ZubBAG4H8BYRNbZbNjNPYeYSZi5p1apVsj9BEALHs1cMxrkD26N7kgqgpEvzqO/sy8rt8QSl0WzRqAANC6L71k6Lph11ac2G2tFHQKrDEq6NRJh5VBLnVAKoVD8vIqINAHoD2Aago+bQjuo2Qcgqjm/XGH++YnDS5z952UDMuPfHyHcbef6yFjdNSyVdm+tuTydzVqDmiRBRKyIKqZ+7A+gFYCMz7wBwkIhOVqOyrgLwkY+iCkJgsNPcxPay02nxI79wOshAqx+M8nF5FdjgBH6F+F5IRFsBnAJgOhHNVHeNALCUiBYDeA/Ajcy8V933KwAvA1gPYAMkMksQUsZOyvFsxY9RQZACDRLhi2OdmT8A8IHO9vcBvG9wTimA/i6LJghZRVBGImlkvUkZKz+1acP0mcQZKHOWIAj2SaUBrk6jZVgzhUQjm2evGJxWmQBEiQhCmnPNqd2SPjcwI5EAmm/OHdje0/LCNVAUE2IcdII4T0QQBBs0Kkj+NU5mBb1M4tP/Ox2Hjumv+fH0ZQPx8IXOW9CN1OWEEzvh09W70K+D7VkNviJKRBCyGDPHer/2jXGeR71xv3wiPVoZL/6UG8pB45Dzxhqj3zq6X1uUPTre8fLcRsxZgpDB6K2Qd/6gesUwopfxZNzptw7HDaf3cEWuMMEzYgl2ESUiCBlAw3x9O7refIMnLx2IZfeNBgA8cekJWHCn/xlhs0mZpNNEQiuIEhGEDGDBXSNRXJCL1sUFUSYRPYd1bigHxWr0T0FuCJ2aF3kmp5B5iE9EEDKA5g3zsez+MXHb06XTm2m982xCRiKCkMFI4yy4jYxEBCFDadIgL2LMeuKSE9CrTbGv8giZiYxEBCEDeWHiEPznltMi5qw+bRtjUKem/gqlwxOXDETHZg3SKuGgEI2MRAQhAxk3oB2Aep9IUNYNieXioR1x8dCOiQ/MQIZ2aea3CI4gSkQQMphwdFa2rWMedD79v9Oj1lpPZ0SJCEIGEzYT1YkWCRRmM+XTDfGJCEImo9qzRIUIbiFKRBAymLC/WgYigluIEhGEDObucX3QqXkDHN9OwnsFdxCfiCBkMCd1b4EFd470WwwhgxElIggCpl5/MrbtO+q3GEIa4os5i4guJaIVRFRHRCWa7ROJaLHmr46IBqn75hHRGs2+1n7ILgiZyMndW2TtfA0hNfwaiSwHcBGAl7QbmflNAG8CABENAPAhMy/WHDKRmUs9k1IQBEEwxRclwsyrgITJ4a4AMNUTgQRBEISkCHJ01gQAb8dse1U1Zd1DJhqIiK4nolIiKi0vL3dXSkEQhCzGNSVCRHOIaLnO3/kWzj0JQAUzL9dsnsjMAwAMV/+uNDqfmacwcwkzl7RqZbz8pyAIgpAarpmzmHlUCqdfjphRCDNvU/8fIqK3AAwD8HoKZQiCIAgpEjhzFhHlALgMGn8IEeUSUUv1cx6An0JxzguCIAg+4leI74VEtBXAKQCmE9FMze4RALYw80bNtgIAM4loKYDFALYB+KtnAguCIAi6+BWd9QGADwz2zQNwcsy2IwCGui+ZIAiCYAfiDM/MRkTlADYleXpLALsdFMcpRC57iFz2ELnskalydWHmhJFJGa9EUoGISpm5JPGR3iJy2UPksofIZY9slytwjnVBEAQhfRAlIgiCICSNKBFzpvgtgAEilz1ELnuIXPbIarnEJyIIgiAkjYxEBEEQhKQRJaIDEY1V1y5ZT0STPC67ExHNJaKV6por/6tuv4+ItmnWUzlHc87dqqxriGiMi7KVEdEytfxSdVtzIppNROvU/83U7UREz6pyLSWiIS7JdFzMGjQHiejXftUXEb1CRLuIaLlmm+06IqKr1ePXEdHVLsn1BBGtVsv+gIiaqtu7EtFRTd29qDlnqPoMrFdlN03FnaRctu+d0++sgVzvaGQqI6LF6nZP6sukbfD3+WJm+dP8AQgB2ACgO4B8AEsA9PWw/HYAhqifiwGsBdAXwH0AfqNzfF9VxgIA3VTZQy7JVgagZcy2xwFMUj9PAvCY+vkcAJ8AICiTR7/x6N79CKCLX/UFJePCEADLk60jAM0BbFT/N1M/N3NBrtEActXPj2nk6qo9LuY636qykir7OBfksnXv3Hhn9eSK2f8kgN97WV8mbYOvz5eMROIZBmA9M29k5iooObwSZh52CmbewczfqZ8PAVgFoIPJKecDmMrMlcz8A4D1UH6DV5wP4DX182sALtBsf50VvgbQlIjauSzLWQA2MLPZ5FJX64uZ5wPYq1OmnToaA2A2M+9l5n0AZgMY67RczDyLmWvUr18DMF3aUJWtMTN/zUpr9LrmtzgmlwlG987xd9ZMLnU0cRnil6qIPc7R+jJpG3x9vkSJxNMBwBbN960wb8Rdg4i6AhgM4Bt1083qsPSV8JAV3srLAGYR0SIiul7d1oaZd6iffwTQxge5wsRmf/a7vsLYrSM/ZLwOSq81TDci+p6IPiei4eq2DqosXshl5955XV/DAexk5nWabZ7WV0zb4OvzJUokoBBRIwDvA/g1Mx8E8AKAHgAGAdgBZTjtNacx8xAA4wDcREQjtDvV3pYv4X5ElA/gPAD/VDcFob7i8LOOjCCiyQBqoC5NDaW+OjPzYAC3A3iLiBp7KFIg752GKxDdWfG0vnTahgh+PF+iROLZBqCT5ntHdZtnkJLu/n0AbzLzvwCAmXcycy0z10HJYBw2wXgmL9ev6bILSgLNYQB2hs1U6v9dXsulMg7Ad8y8U5XR9/rSYLeOPJORiK6BsrTCRLUBgmou2qN+XgTF39BblUFr8nJFriTunZf1lQvgIgDvaOT1rL702gb4/HyJEolnIYBeRNRN7d1eDmCaV4Wr9ta/AVjFzE9ptmv9CReifj2VaQAuJ6ICIuoGoBcUZ57TcjUkouLwZyhO2eVq+eHojqsBfKSR6yo1QuRkAAc0Q243iOod+l1fMdito5kARhNRM9WUM1rd5ihENBbAnQDOY+YKzfZWRBRSP3eHUkcbVdkOEtHJ6nN6lea3OCmX3Xvn5Ts7CsBqZo6YqbyqL6O2AX4/X8l65DP5D0pUw1ooPYrJHpd9GpThaHjtlMWqPG8AWKZunwagneacyaqsa5BitIyJXN2hRL0sAbAiXC8AWgD4FMA6AHMANFe3E4DnVbmWAShxsc4aAtgDoIlmmy/1BUWR7QBQDcXW/Itk6giKj2K9+netS3Kth2IbDz9nL6rHXqze48UAvgNwruY6JVAa9Q0AnoM6YdlhuWzfO6ffWT251O1/B3BjzLGe1BeM2wZfny+ZsS4IgiAkjZizBEEQhKQRJSIIgiAkjSgRQRAEIWlEiQiCIAhJI0pEEARBSBpRIoJgABHVUnSGYNPssER0IxFd5UC5ZUTUMonzxhDR/aRkdf0k8RmCkDq5fgsgCAHmKDMPsnowM7+Y+ChXGQ5grvr/C59lEbIEGYkIgk3UkcLjpKwT8S0R9VS330dEv1E/30rKug9LiWiquq05EX2obvuaiE5Qt7cgolmkrBHxMpRJYuGyfq6WsZiIXgrPjI6RZwIpa1vcCuAZKKlCriUizzItCNmLKBFBMKZBjDlrgmbfAWYeAGUW8jM6504CMJiZTwBwo7rtfgDfq9t+CyU1OADcC+ALZu4HJSdZZwAgouMBTABwqjoiqgUwMbYgZn4HSkbX5apMy9Syz0vlxwuCFcScJQjGmJmz3tb8f1pn/1IAbxLRhwA+VLedBiVFBpj5M3UE0hjKAkgXqdunE9E+9fizAAwFsFBJm4QGqE+uF0tvKIsLAUBDVtabEATXESUiCMnBBp/DjIeiHM4FMJmIBiRRBgF4jZnvNj1IWaq4JYBcIloJoJ1q3rqFmRckUa4gWEbMWYKQHBM0/7/S7iCiHACdmHkugLsANAHQCMACqOYoIjoDwG5W1oOYD+Bn6hSwdDQAAADZSURBVPZxUJYsBZSkepcQUWt1X3Mi6hIrCDOXAJgOZSW7x6EkIBwkCkTwAhmJCIIxDdQefZgZzBwO821GREsBVEJJQ68lBOAfRNQEymjiWWbeT0T3AXhFPa8C9em77wfwNhGtAPAlgM0AwMwrieh3UFaTzIGSUfYmAHrL/w6B4lj/FYCndPYLgitIFl9BsAkRlUFJq73bb1kEwW/EnCUIgiAkjYxEBEEQhKSRkYggCIKQNKJEBEEQhKQRJSIIgiAkjSgRQRAEIWlEiQiCIAhJI0pEEARBSJr/D5XdflrV7i6XAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def ddpg(n_episodes=2000, max_t=700):\n",
    "    scores_deque = deque(maxlen=100)\n",
    "    scores = []\n",
    "    max_score = -np.Inf\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        state = env.reset()\n",
    "        agent.reset()\n",
    "        score = 0\n",
    "        for t in range(max_t):\n",
    "            action = agent.act(state)\n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "            agent.step(state, action, reward, next_state, done)\n",
    "            state = next_state\n",
    "            score += reward\n",
    "            if done:\n",
    "                break \n",
    "        scores_deque.append(score)\n",
    "        scores.append(score)\n",
    "        print('\\rEpisode {}\\tAverage Score: {:.2f}\\tScore: {:.2f}'.format(i_episode, np.mean(scores_deque), score), end=\"\")\n",
    "        if i_episode % 100 == 0:\n",
    "            torch.save(agent.actor_local.state_dict(), 'checkpoint_actor.pth')\n",
    "            torch.save(agent.critic_local.state_dict(), 'checkpoint_critic.pth')\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))   \n",
    "    return scores\n",
    "\n",
    "scores = ddpg()\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(1, len(scores)+1), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Watch a Smart Agent!\n",
    "\n",
    "In the next code cell, you will load the trained weights from file to watch a smart agent!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent.actor_local.load_state_dict(torch.load('checkpoint_actor.pth'))\n",
    "agent.critic_local.load_state_dict(torch.load('checkpoint_critic.pth'))\n",
    "\n",
    "state = env.reset()\n",
    "agent.reset()   \n",
    "while True:\n",
    "    action = agent.act(state)\n",
    "    env.render()\n",
    "    next_state, reward, done, _ = env.step(action)\n",
    "    state = next_state\n",
    "    if done:\n",
    "        break\n",
    "        \n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Explore\n",
    "\n",
    "In this exercise, we have provided a sample DDPG agent and demonstrated how to use it to solve an OpenAI Gym environment.  To continue your learning, you are encouraged to complete any (or all!) of the following tasks:\n",
    "- Amend the various hyperparameters and network architecture to see if you can get your agent to solve the environment faster than this benchmark implementation.  Once you build intuition for the hyperparameters that work well with this environment, try solving a different OpenAI Gym task!\n",
    "- Write your own DDPG implementation.  Use this code as reference only when needed -- try as much as you can to write your own algorithm from scratch.\n",
    "- You may also like to implement prioritized experience replay, to see if it speeds learning.  \n",
    "- The current implementation adds Ornsetein-Uhlenbeck noise to the action space.  However, it has [been shown](https://blog.openai.com/better-exploration-with-parameter-noise/) that adding noise to the parameters of the neural network policy can improve performance.  Make this change to the code, to verify it for yourself!\n",
    "- Write a blog post explaining the intuition behind the DDPG algorithm and demonstrating how to use it to solve an RL environment of your choosing.  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
